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The purpose of this paper is to explore: a) the nature and status of the mathematics 
reform movement in America’s public schools, b) the connection between the reform 
movement (focusing on California initiatives) and constructivist epistemology, c) the 
development of an assessment tool for measuring the degree of reform present in a 
secondary mathematics classroom, and d) the potential for investigating the relationship 
between the degree of reform and student achievement. 

Student performance in mathematics at the secondary level has been an issue of great 
concern to American educators, parents, and government officials, especially in the last 1 5 
years. Declining student scores in mathematics in the United States and poor national 
performance in comparison to other countries have led to a widespread call for a reform of 
public education (National Commission on Excellence in Education, 1983; National 
Research Council, 1989). With the publication of A Nation at Risk by the National 
Commission on Excellence in Education (1983), the Curriculum and Evaluation 
Standards by the National Council of Teachers of Mathematics (NCTM) in 1989, and the 
California Department of Education’ sMa//iema//c 5 Framework (1992), a new reform 
platform was established in California for mathematics education. The new platform 
included: (a) the identification of the content to be taught into 14 mathematics standards, 
(b) recommendations about instructional strategies and student activities, and (c) a new 
underlying philosophy about how learning happens in the mathematics classroom. 

Despite these publications, student performance in California has continued to fall 
behind that of most other states and developed countries (California Department of 
Education, 1995). Recent results fi-om the Third International Mathematics and Science 
Survey (TIMSS) indicate that mathematics achievement scores for students in the United 
States are below levels in other industrialized countries (U.S. Department of Education, 
1996). These declining scores have been used by some as a justification for a call to 
return to a traditional approach to teaching mathematics, focusing on the acquisition of 
skills and memorization of facts. Others favor the reformed approach, based on the 
NCTM Standards and the California Mathematics Framework (1992). California State 
Superintendent of Public Schools Delaine Eastin charged a Task Force in April 1995 to 
make recommendations to “do what needs to be done to turn education in mathematics 
around” (California Department of Education, 1995, p. 1). The California Department of 
Education issued the California Mathematics Task Force Report to the Superintendent on 
September 13, 1995. Included in the report are the recommendations of the Task Force 
for improving mathematics achievement in California schools (California Department of 
Education, 1995, p. 3). These recommendations include a call for: 

1 . The establishment of clear and specific content and performance standards for 

mathematics and support of districts and schools to make these standards achievable by all 
students. 
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2. The establishment of a stable, coherent, and informative system of assessment for 
all California students. 

3. The guarantee of high quality mathematics instruction by providing adequate 
time, instructional materials, mathematically powerful teachers, and time for teacher 
collaboration. 

4. The establishment of a management, research and information system to answer 
basic questions about the operation and effectiveness of mathematics policies and 
programs, including implementation and effectiveness of the stale’ s Mathematics 
Framework. 

5. The identification of responsibilities for the school and the home to support the 
parents’ role in their children’s education. 

While the Department of Education has initiated activities related to the first two 
recommendations, the latter three recommendations have received little attention to date. 
Of particular interest is Recommendation 4, calling for research to improve mathematics 
education. The state Mathematics Framework (1992) describes the new platform for 
mathematics education, but has no system for determining whether it is effective, or even 
whether the reforms are being implemented in California classrooms. The Task Force 
recommended that a new program be established to monitor the implementation and 
effectiveness of educational policies. The function of such a program would be to 
undertake systematic observations of mathematics classrooms in action and studies of 
student achievement, in order to determine which programs and policies are effective. 

The purpose of this study was to develop and assess the predictive validity of an 
instrument measuring the degree to which secondary mathematics classrooms have 
implemented the policies, strategies, and philosophy of the California Framework (1992), 
and the NCTM Standards (1992). The instrument will utilize questionnaire data and 
videotapes of classroom instruction. The degree of reform will be assessed along the 
following four areas; (a) teacher beliefs, (b) teacher activities, (c) student activities, and 
(d) curricular content and methodology. 

The Nature Of The Mathematics Reform Movement 

The California Framework describes reformed mathematics classrooms by describing 
the abilities of learners: “Mathematically powerful students think and communicate, 
drawing on mathematical ideas and using mathematical tools and techniques” (California 
Department of Education, 1992, p. 20). The four dimensions of mathematical power are 
then described as mathematical thinking, communication, drawing upon mathematical 
ideas, and the use of mathematical tools and techniques. These dimensions parallel the 
process standards in the NCTM Curriculum and Evaluation Standards (1989); 

1 . Mathematics as Problem Solving 

2. Mathematics as Communication 

3. Mathematics as Reasoning 

4. Mathematical Connections 
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The reformed mathematics described in each of the documents offers guidelines in the 
arenas of curricular materials, instructional strategies, and student activities. Underlying 
the recommendations of both documents is a constructivist philosophy of how learmng 
occurs. 

Constructivism as a model of learning has become a popular theoretical basis for many 
educational reform decisions made in Western societies. The tenets of constructivism 
have served as a basis for the beliefs of many teachers and researchers about the way 
classrooms should operate. The National Research Council declares. Research in 
learning shows that students actually construct their own understanding based on new 
experiences that enlarge the intellectual framework in which ideas can be created” (1989, 
p. 6). The California Framework (1992) affirms the research of Piaget and 

Vygotsky, stating that students are not passive absorbers of knowledge, but rather 
constructors of their own understanding of mathematics. The Framework stresses that 
students learn mathematics by using it to make sense of their own experience. This 
learning is made more powerful when students are able to use it “to achieve purposes that 
are meaningful to them” (California Department of Education, 1992, p. 33). 

The Process of Making Meaning 

Analysts of the knowledge construction process have outlined several component 
activities typically present in developing meaning in mathematics: Encountering a new 
situation, comparing the new situation with prior knowledge, hypothesis creation and 
testing, and reflection. 

Encountering a new situation . Knowledge construction begins when the learner 
encounters a new situation (Appleton, 1993). Tobin (1989) calls this first step sense 
“perception.” Data come into the mind through the physical senses. Often this is where 
most education stops, as many objectivist-oriented educators believe that knowledge has 
been acquired (Etchberger & Shaw, 1992; Roth, 1993). These data can come in many 
forms: a new experience encountered, a new problem posed, or a simple observation. 

Then the data are checked against prior knowledge. 

Comparing the new situation with prior knowledge . After the sense perception and 
analysis the learner may, appropriately or not, assimilate this experience along with some 
prior knowledge. If the data do not fit into an existing cognitive framework however, a 
state of perturbation is entered. Several theorists have described this part of the learning 
cycle as the key experience in learning, calling it by such names as dissonance (Festinger, 
1957), disequilibrium (Piaget, 1978), mini-theory testing (Claxton, 1990), or perturbation 
(von Glasersfeld, 1987). Cobb (1994b) writes, “perturbations that the cognizing subject 
generates relative to a purpose or goal are posited as the driving force of development” (p. 
14). This perturbative stage is where the individual becomes motivated to gather more 
data and to create and test hypotheses. Fosnot (1989) calls this process disequilibration, 
following Piaget’s terminology, and seeks to activate this sense in her students by selecting 
new situations which will cause them to feel discomfort. 
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Hypothesis creation and testinjz . Whether the new situation was assimilated into a 
previously acquired knowledge scheme or a new hypothesis was created, the mind 
investigates the viability of the thinking generated (Fosnot, 1989; von Glasersfeld, 1995a). 
The learner often concludes that this knowledge is relevant to the task at hand and can be 
used to arrive at a solution (Davis & Maher, 1990). This action to resolve disequilibrium 
may involve utilizing multiple approaches (Confrey, 1995). 

Reflection . The next stage is an analytical one described by many theorists as 
reflection (Etchberger & Shaw, 1992; von Glasersfeld, 1987; Wheatley, 1992). The 
learner is engaged in making sense of the gathered data and his/her representation of the 
solution process. Reflection is described as distancing oneself from the act of doing and 
looking at one’s own actions. This can be done individually or socially (Cobb, 1994b). 
Reflection typically has three evaluative components: (a) The interpretation of the 
problem, (b) a review of the viability of the process used, and (c) justification of 
successful strategies (Confi-ey, 1990). Piaget termed this process "reflective abstraction" 
and it is interesting to note that two alternate definitions of "reflection" are both 
appropriate to the work done by the learner: reflection as “pondering” and reflection as 
“bouncing my ideas off* of other minds for feedback” (von Glasersfeld, 1995b). During 
reflection, one constructs schemes of schemes, asking “Will my approach work?” Often 
the answer is negative and the learner must continue in disequilibrium. Here, or at several 
other stages, the learner may also elect to opt out of learning (Appleton, 1993). 

If the analysis proves that the learner’s hypothesis is correct, or illuminates another 
more efficient hypothesis, the mind then adapts internal cognitive structures to 
accommodate the new conjecture. Satisfied, the perturbation is eliminated and equilibrium 
returns. 

Because most classrooms today are operating from the assumptions of objectivist or 
behaviorist theories, determining how to change the classroom to reflect the reform based 
on constructivist philosophy is an important task. Priority needs to be given to how a 
student is constructing knowledge, with less emphasis on information being delivered by 
the instructor. 

One note of caution must precede discussion of classroom characteristics that support 
constructivist theory. Cognizing minds construct knowledge regardless of how 
experiences are encountered. Whether a student is presented with an authentic experience 
or a worksheet with 50 arithmetic problems, the student will use the processes of 
accommodation and assimilation to reestablish equilibrium. Students construct knowledge 
even in the most authoritarian settings. Constructivism is a theory of how knowledge is 
constructed, not necessarily a handbook for classroom practice. Research must focus on 
the nature and quality of the constructions and which classroom environments best 
contribute to quality knowledge construction, rather than whether or not knowledge 
construction is occurring (Cobb, 1994a). 
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Classroom Applications of Constructivist Epistemology 

Recommendations for applying constructivist ideas to the classroom are centered 
around three broad themes; Teacher beliefs and activities, curricular prompts and 
activities, and student processes and activities. 

Reform recommendations about teacher beliefs and activities. Inspection of the 
literature reveals that many of the guiding principles of the Framework (1992) about the 
activities of the teacher have been informed by researchers of a constructivist-based 
philosophy of learning. Teacher beliefs, conceptions and personal theories about subject 
matter, teaching, and learning all affect the creation of a constructivist environment 
(Ernest, 1995). The quality of student constructions depends in large part on the 
assumptions and activities of the teacher (Carroll & Balfanz, 1995). 

The Mathematics Framework (1992) devotes an entire section to the role of the 
teacher in creating mathematically powerful classroom environments. The optimal 
atmosphere has students taking responsibility for understanding, doing viable work, 
managing time resources, and communicating with others. Teachers are to assume the 
roles of guide and pacer, and need to model what it looks like to be a practicing 
mathematician. 

The Framework's guiding principles for teaching for understanding are as follows: 

1 . Development of student thinking and understanding. 

2. Creating direct personal experiences in which students can verify or alter their own 
understanding, rather than depend on an authority. 

3. Deep understanding takes time; teachers need to provide many varied opportunities 
to explore a particular concept. 

4. Students learn at many levels and rates. Teachers must provide activities that can 
be understood at many levels. 

5. Students need to work with each other in small groups. 

6. Periods of confusion are an important part of learning; teachers must resist the 
temptation to give immediate answers and instead frame questions to further investigation. 

7. Teachers must be students of the thinking process that learners are going through. 

8. Students must be allowed to reflect upon the ideas found in activities. 

9. Teachers should foster a questioning attitude in students. 

10. Teachers need to recognize the importance of student verbalization. 

(California Department of Education, 1992, pp. 52-53) 

Pirie and Kieren (1992) record four underlying tenets of belief necessary to creating a 
constructivist mathematics environment; 

1 . The teacher must recognize that particular learning goals will be achieved at 
differing rates and levels by the students; the teacher must be continually recreating 
situations which reflect current student constructions. 
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2. The teacher must recognize that there are multiple pathways to similar 
mathematical understanding; each child comes to his/her current level of understanding 
through an uniquely individual pattern of engagement with the learning content. 

3. The teacher will be aware that students hold different mathematical 
understandings. 

4. The teacher will know that levels of understanding are never reached once and 
for all. 

Implicit in these tenets is the belief that there is no common mathematical 
understanding waiting “out there” to be transmitted. Understanding is developed rather 
than transmitted. Teachers must then encourage the development of metaphors and 
mathematical representation systems (Brooks & Brooks, 1993; Cobb, Wood, Yackel, & 
Perhvitz, 1992). 

A critical activity of the teacher in the constructivist environment is creating situations 
in which students actively participate in activities that encourage them to make their own 
individual constructions (Wood, 1995). Teachers can encourage these activities by asking 
thoughtful, open-ended questions (Brooks & Brooks, 1993) and by placing the focus of 
activity on student interaction with the problem, not teacher presentation of a skill 
(Serrano & Stone, 1995). 

In an environment marked by constructivist philosophy, the teacher must seek to 
understand and value the students’ point of view (Confrey, 1990; Ernest, 1995; Wood, 
1995). Confrey suggested that throughout the learning process teachers must build 
models of students’ understanding of mathematics (Confrey, 1990). She described these 
models as “case studies” for each student. The perspective that a student brings to an 
encounter is seen as an instructional entry point in the learning experience. Thus, the 
process of learning is not centered around having the students focus on what the teacher 
knows, but rather the teacher focusing on what the student knows and creating 
experiences which will extend and challenge the student. Asking students to reflect on 
their thinkin gs or elaborate on their solution to a problem, allows the teacher to make 
assessments about how students view a situation. Rather than focusing on “errors in 
thinking, ” teachers see student approaches as an indication of current understanding (von 
Glasersfeld, 1995a). Focusing on student points of view also enables the teacher to 
enq)loy another principle, adapting curriculum to address student suppositions. Learning 
is enhanced when the demands of the curriculum have a direct relationship to student 
suppositions (Brooks & Brooks, 1993; Carroll & Balfanz, 1995). 

In the typical mathematics classroom created to foster knowledge construction, many 
students will each be building their own representation of the problem and its solution. 
Processing the problem posed will not rely on teacher prescription, but on several of the 
representations invented by students (Jonassen, 1991). Teachers will need to build much 
more time into problem analysis and encourage multiple representations and explanations 
(Merseth, 1993). 

Brooks and Brooks (1993) believe that assessing student learning in the context of 
teaching is imperative to a constructivist environment. Rather than simply monitoring the 
body of facts that a student has committed to recall, assessment embodies the much 
broader activities of viewing how the student is constructing knowledge. 
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Reform recommendatiotis about written curriculum . In addition to teacher practices, 
the curriculum itself can reflect reformed mathematics principles. The Mathematics 
Framework (1992) recommends that written curriculum be created as an instructional unit . 
A powerful instructional unit shows much more depth than a collection of lessons and 
activities. The instructional unit described by the Framework is coherent, with a clearly 
formulated purpose. All activities are related to a primary goal, and developed in an 
engaging, meaningful context; 

Each coherent piece interweaves strands, ensuring breadth, and each deepens one 
or more unifying ideas. In each, students will do mathematics and be responsible 
for the thinking. They will investigate rich and complex problem situations, 
conjecture, explore ideas, make cormections among mathematical ideas, and 
generalize their findings. (California Department of Education, 1992, pp. 90-91) 

Curriculum units are recommended to be about 6 weeks long for secondary students, 
focusing on large assignments (multi-day), open-ended problems, and exercises which lead 
toward developing the main idea in the unit. Again, the Framework's view of curriculum 
that supports powerful learning of mathematics can be found in the research on 
constructivist learning. 

Curricula created to foster knowledge construction require instructors to pose 
problems of emerging relevance (Brooks & Brooks, 1993). Students do not need to have 
prior interest or complete autonomy in choosing topics of study for the relevance principle 
to be met. Relevance can emerge as teacher and students interact with a problem that 
demands the generation of a hypothesis, has multiple approaches and is complex. 
Relevance is considered to be an important factor in student motivation according to 
Keller’s ARCS model of motivation (Bohiin & Bohlin, 1995). Furthermore, students who 
are experiencing perturbance with respect to a new situation will find the problem solving 
process to be relevant to their own understanding. Various researchers have stressed the 
need for relevant curricula which will encourage teacher use of raw data and primary 
sources (Brooks & Brooks, 1993; Roth, 1993). 

Another principle of a constructivist curriculum is to structure learning around primary 
concepts. Brooks and Brooks (1993) state that students are more engaged when 
problems are presented holistically than when component skills are taught in isolation fi'om 
each other. Constructed knowledge must revolve around an essential idea. Other related 
characteristics of the principle of presenting big ideas are their relevance and openness for 
multiple approaches. 

Constructivist curricula employ discovery learning as a typical approach to 
investigation of topics. Davis (1990) names discovery-oriented curricula as one of the 
effective practices of the “New Math” movement in the late 1950s through the 1970s. In 
discovery learning, each activity is plaimed to guide the student to draw his/her own 
conclusions, to make generalizations and to develop the mathematical concepts necessary 
to solve the larger problem. Students are required to communicate their ideas in the 
micro-society of the classroom, presenting, explaining and justifying the mathematics that 
they have developed. 
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Reform recommendations about student activities . In addition to teacher behaviors 
and curricular formats, there are several activities unique to the learners themselves that 
are indicative of a reformed mathematics classroom. One of these activities revolves 
around the concept of student autonomy and initiative (Brooks & Brooks, 1993; Roth, 
1993; Taylor & Fraser, 1991). If students create their own interpretations of mathematics 
regardless of the environment (Cobb, 1994b; Wood, 1995), then it stands to reason that 
they should exercise control over decisions about their own learning process. Case studies 
of teachers who exhibit constructivist ideas reveal many occurrences of student autonomy 
and initiative (Clarke, 1995; Confrey, 1990; Roth, 1993; Wood, Cobb, & Yackel, 1995). 
Measurement tools of the constructivist environment include scales for student decision 
making with respect to all classroom practices (Taylor & Fraser, 1991). Case studies of 
teachers in training reveal that student self-regulation is actually a component of 
constructivism that teachers discover through teaching experiences (Fosnot, 1 989). These 
researchers all indicate that student self-assessment and decision making about how to 
proceed is more effective than teacher advice on “how to do it.” 

An equally important student activity is the use of student dialogue in the reflection 
process (Brooks & Brooks, 1993; Cobb, 1994b) . Instruction is interactive, involving 
both student-to-student and student-to teacher-dialogue (Cobb, Wood, & Yackel, 1990; 
Confrey, 1990). Driver (1995) writes that discussion with peers is an important element in 
knowledge construction. Discussion provides an arena for making implicit ideas explicit. 

It allows individuals to clarify their thinking, and offers the opportunity for students to 
build from each other’s ideas a deeper understanding of the concepts involved. 

Through dialogue and student decision making, students have the opportunity to 
examine the viability of their personal constructions. Students therefore have the 
opportunity to adapt their constructions when they see misconceptions. Several 
researchers have attested to the tenacity with which children hold to naive views of 
problem situations (Bodner, 1986; Merseth, 1993; von Glasersfeld, 1995a). Case studies 
reveal that teacher-provided information contrary to these misconceptions does little to 
alter t hinkin g. Von Glasersfeld (1995a)writes: 

M long as the counterexamples provided by the teacher are taken from areas that 
lie outside the students’ field of experience, they are unlikely to lead to a change in 
the students’ thinking. Only when students can be led to see as their own a 
problem in which their approach is manifestly inadequate will there be any 
incentive for them to change it. (p. 15) 

Dialogue is necessary in other important components of the learning process: (a) 
Elaboration of initial responses (Brooks & Brooks, 1993), (b) Cooperative learning 
(Etchberger & Shaw, 1992), and (c) Consensus building among student groups 
(Etchberger& Shaw, 1992). 
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The Purpose of Mathematical Constructions 

The goal of student, teacher, and curricular activities is powerful student mathematical 
construction. Powerful constructions are characterized by Confrey (1990) as having the 
following characteristics; 

1 . A structure with a measure of internal consistency. 

2. Integration across a variety of concepts. 

3. Convergence among multiple forms and contexts of representation. 

4. An ability to be reflected on and described. 

5. Historic continuity. 

6. Ties into various symbol systems. 

7. Agreement with experts. 

8. A potential to act as a tool for further constructions. 

9. A guide for future actions. 

10. An ability to be justified and defended. 

Measuring the Degree of Reform in Mathematics Classes 

Reformed instruction and constructivist practices have become benchmarks for quality 
reformed mathematics learning among leading educators and theoreticians. However, 
very little has been done to quantify the effects of a reformed environment on student 
performance. There are several obstacles to be overcome before quality research can be 
accomplished regarding the effectiveness of these environments. 

If constructivist pedagogy is to be used as a basis for defining effective instruction as 
an independent variable, then a coherent, well-planned instrument must be developed to 
discriminate environments as either mathematically reformed or traditional. None of the 
instruments currently available incorporate all four elements described here: curriculum, 
teacher beliefs, teacher activities, and student activities. Any instrument to be developed 
must be tested before classrooms can be validly and reliably identified along a 
reformed/traditional continuum. 

Finally, the issue of how to properly assess students who learn in a reformed 
environment has also been raised. Along with the call for reform in mathematics 
education, there has been a call for reform in assessment. Many educators believe that 
traditional textbook and nationally-normed tests have served as profound barriers to 
educational reform. Because fiU-in-the-blank and multiple-choice tests do not provide 
meaningful measures of higher-order thinking and reasoning skills, these skills are not 
valued in the typical classroom (Merseth, 1993). 

Determining what types of assessment tools best measure student achievement is a 
hotly debated educational and political issue. Although authentic assessment instruments 
are being developed, there is some doubt about the validity and reliability of these 
instruments. It does not seem prudent or appropriate however, to rely on standard 
multiple-choice assessments to determine the effectiveness of knowledge construction. 
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Methodology 

What follows in this paper is the presentation and analysis of an instrument to measure 
specifically where secondary mathematics classrooms fall on a traditional/reformed 
continuum. The instrument utilizes a video analysis inventory to measure observed 
behaviors, and a questionnaire to measure teacher beliefs about the classroom 
environment. 

The Instrument 

The two-part measurement instrument which was developed to be used in the study 
consists of a teacher beliefs questionnaire, and a video analysis inventory, named the 
Secondary Mathematics Classroom Reform Inventory (SMCRI), which is divided into 
three subscales. The three subscales of the SMCRI, and the Teacher Beliefs Questionnaire 
comprise the four independent variables related to the areas of classroom characteristics 
described in the literature review: 

1 . The Teacher Beliefs Questionnaire contains 3 1 items probing teachers’ beliefs about 
how learning should take place and actual instructional practices reported by the teacher. 
The questions were designed to avoid influencing the teacher either toward a reform 
perspective or a traditional philosophy of instruction. A summary score was given for 
each teacher, indicating placement along the continuum from reformed to traditional 
philosophy and practice. The response scales are Likert five-point scales, with a score of 
1 indicating little or no reform, and a score of 5 indicating a high degree of reform 

2. The Secondary Mathematics Classroom Reform Inventory (SMCRI') was developed to 
rate how fully classroom activities and prompts are occurring in the classroom. The 
SMCRI was used by the classroom/video observer, in this case the researcher. The 
response scale for all items in the SMCRI is a five-point Likert scale, with 1 = Completely 
absent and 5 = Fully emphasized. The three subscales on the SMCRI are. 

A The SMCRI Teacher Activities Subscale, containing 14 items assessing the nature 
of what teachers do to enable student processing, communication, and reflection. 

B. The SMCRI Curricular Activities Subscale, containing 9 items indicating degree of 
curricular complexity, and identifying the sources and prompts used. 

C. The SMCRI Student Activities Subscale, containing 9 items observing student 
autonomy, communication, and cognitive functions. 

A fifth variable used in the study was the expert holistic rating of the degree to which 
the classroom exhibits the characteristics of reform. This rating serves as the dependent 
variable in the study. For this variable, overall holistic ratings independent of the SMCRI 
were provided by an expert in reformed mathematics education for the videotapes of the 
mathematics lesson. This holistic rating used the following 5-point rubric: 

1 = No evidence of reformed curriculum and instructional practices 

2 = Little evidence of reformed curriculum and instructional practices 

3 = Reformed curriculum and instructional practices are somewhat evident 




4 = Reformed curriculum and instructional practices are largely evident 

5 = Reformed curriculum and instructional practices are completely evident 

Procedure 

Teachers from two high schools in Central California were invited to participate in the 
study. They were informed that the study involved observation of various teacher and 
classroom characteristics, and that the study also sought to compare teacher beliefs with 
instructional practices. Two of the 17 teachers who were asked to partfcipate declined. 
Teachers were asked to select a class period and course that best represents their teaching 
style. These 15 mathematics classrooms were videotaped for one period during the month 
of February in the 1995-96 school year. The Teacher Beliefs Questionnaire was filled out 
by the same teachers approximately 2 weeks prior to the videotaping and returned to the 
researcher. 

The classrooms reflect various socioeconomic communities and diverse student ethnic 
populations. Both high schools are populated with students ranging from low- to middle- 
socioeconomic status. Both schools have approximately the same socio-ethnic 
populations: 45% Hispanic, 30% Euro-American, 10% African American, and 15% 
Asian-American. Both schools receive Tittle 1 funding because a certain percentage of 
the student population demonstrates standardized test scores below the 45% percentile. 
The courses videotaped include: Math A, Algebra 1, Geometry, Algebra 2, Interactive 
Mathematics 1 and Interactive Mathematics 3. The Interactive Mathematics Program is a 
National Science Foundation sponsored curriculum project intended to reflect the NCTM 
Standards. 

The videotapes focused on the teacher vocalizations, student vocalizations, and 
various other activities in the classroom. The videos were taped by high school students. 
The tapes focus on the instructor during all lecture periods, and on particular groupings of 
students who were either engaged in the activity, or working directly with the teacher 
during student activities. In addition, a copy of the written prompt/text for the lesson was 
provided to the researcher. Evaluations along the Curricular Scale utilized both the 
written prompt and the activities of both teachers and students. 

The videos were observed by the researcher using the SMCRI, then by mathematics- 
educator-experts who had received training in the teaching of reformed mathematics for at 
least 100 hours over the past five years. The training included either participation in the 
San Joaquin Valley Mathematics Project or in inservicing for a particular reformed 
mathematics curriculum. Each of the 1 5 videotapes was viewed and rated by three 
experts. Each expert viewed a different combination of tapes. Interrater reliability was 
established by determining the percent of ratings given which match at least one other 
expert rating. If the percent agreement met or exceeded 67% (four of six ratings), then 
the expert was considered reliable. Once interrater reliability was established, the mean of 
the ratings of the reliable experts was calculated and used as the dependent variable. 
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Statistical Analyses 

The usefulness of the instrument was determined by analyzing its predictive validity. 
Means and sample standard deviations were computed for each of the individual items on 
the instrument for descriptive purposes. Means and summary scores were computed for 
each of the teachers on each subscale in order to determine if the subscales produced a 
range of scores. Correlations were computed using the summary score for each of the 
independent variables, the sum of the video subscales, the sum of the video and 
questionnaire subscales, and the expert rating. The Pearson correlation coefficient was 
used to determine the degree of interrelatedness between independent variables, and the 
degree of relatedness between single subscales and the expert rating. Multiple linear 
regression was performed to determine the predictive validity of the instrument in 
determining the degree of reform evident in a particular classroom setting. 

Results 

Tables 1 and 2 summarize the data collected from the Teacher Beliefs Questionnaire 
and the Secondary Mathematics Classroom Reform Inventory. The data from the 
questionnaire is presented in Table 1 , showing means and teacher summary scores for the 
subscale. The video data is presented in Table 2, showing each of the SMCRI video 
subscales, and teacher summary scores for each of the subscales. The means provide 
information about the center of the responses on the five-point response scale, while the 
standard deviation offers insight into the variability seen in the responses from the 1 5 
instructors. All 1 5 teachers responded to each of the items in the teacher beliefs 
questionnaire, and data was collected for each teacher for every item on the classroom 
inventory form. Therefore, n = 1 5 in all tables. 

Teacher Beliefs Questionnaire and SMCRI Summary Scores 

Teacher Beliefs Questionnaire . The teacher questionnaire responses were added 
together, and a mean was computed for each teacher. Table 1 contains the raw score total 
and mean for each of the teachers. The teachers are identified simply as teacher #1 
through #15. The mean raw score for the teachers in the study was 113, with a standard 
deviation of 20. 1 . The mean of the teachers’ averages was 3 .64, with a standard deviation 
of 0.65. Eleven of 15 (73%) teachers in this study self-report their teaching methods and 
philosophy to be between 2 and 4, which is somewhere between partially reformed and 
largely reformed on the traditional/reformed continuum. 

SMCRI . One research question to be addressed in the study refers to the ability of the 
SMCRI to indicate an acceptable range of summary scores for each of the subscales. 

Table 2 presents the teacher summary score for each of the three video subscales, as well 
as the mean score on that subscale. The Teacher Activities subscale scores ranged from 
1 .0 to 4. 1 , with a mean of 2.39. The Curricular Activities subscale exhibited a range from 
1.0 to 5.0, with a mean of 3.15. 
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Table 1 

Summary Scores and Means bv Teacher on the Teacher Beliefs Questionnaire 



Teacher Questionnaire Totals by Teacher 


Sum of Item Mean 

Responses 


Teacher #1 


107 


3.5 


Teacher #2 


116 


3.7 


Teacher #3 


83 


2.7 


Teacher #4 


120 


3.9 


Teacher #5 


126 


4.1 


Teacher #6 


114 


3.7 


Teacher #7 


125 


4.0 


Teacher #8 


87 


2.8 


Teacher #9 


88 


2.8 


Teacher #10 


90 


2.9 


Teacher #11 


96 


3.1 


Teacher #12 


144 


4.6 


Teacher #13 


151 


4.9 


Teacher #14 


143 


4.6 


Teacher #15 


121 


3.9 
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Table 2 

Sums and Means on the Secondary Mathematics Classroom Reform Inventory bv Each 
Teacher on the Teacher. Curricular, and Student Subscales 



Teacher Scores Teacher Curricular Student 





Sum 


Mean 


Sum 


Mean 


Sum 


Mean 


Teacher#! 


30 


2.1 


29 


3.2 


22 


2.4 


Teacher #2 


53 


3.8 


44 


4.9 


32 


3.6 


Teacher #3 


28 


2.0 


10 


1.1 


13 


1.4 


Teacher #4 


24 


1.7 


21 


2.3 


12 


1.3 


Teacher #5 


41 


2.9 


38 


4.2 


31 


3.4 


Teacher #6 


38 


2.7 


40 


4.4 


22 


2.4 


Teacher #7 


24 


1.7 


22 


2.4 


25 


2.8 


Teacher #8 


38 


2.7 


31 


3.4 


25 


2.8 


Teacher #9 


17 


1.2 


12 


1.3 


11 


1.2 


Teacher #10 


14 


1.0 


11 


1.2 


9 


1.0 


Teacher #1 1 


26 


1.9 


9 


1.0 


12 


1.3 


Teacher #12 


48 


3.4 


42 


4.7 


30 


3.3 


Teacher #13 


32 


2.3 


44 


4.0 


20 


2.2 


Teacher #14 


58 


4.1 


45 


5.0 


36 


4.0 


Teacher #15 


31 


2.2 


27 


3.0 


12 


1.3 
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The Student Activities subscale displayed a range from 1 .0 to 4.0, with a mean of 2.3 1 . 

The data suggest that this sample produced a wide range along the possible continuum 
between traditional and reformed classroom environments. 

Correlation of Variables 

Table 3 presents results of a correlational analysis which was conducted between the 
four independent variables, the video summary score, the video plus questionnaire 
summary score, and the expert mean for each of the 1 5 subjects. All of the combinations 
of variables had a high degree of correlation with the lowest being r = .612 and all of the 
correlations were significant at the p < .01 5 level. The highest correlation was between 
the Teacher Activity subscale and the summary score for the questionnaire and the video 
subscales combined, r = .958. This value is inflated because the Teacher Activity subscale 
score is included in this particular summary score as well. Each instrument subscale 
correlated highly with the expert rating as well, the lowest r = .757, and all were 
significant at the p < .001 level 

Regression Analysis 

Multiple linear regression analysis was performed using the summary score for the 
Teacher Beliefs Questionnaire, and each of the three SMCRI video subscales as the four 
independent variables. The dependent variable was the mean of the three reliable expert 
ratings. Table 4 summarizes the analysis. 

The resulting predictive equation is as follows: 

Y’ = -1.569 + .016(Tchr. Q.) - .031(Tchr. Act.) + .026(Curr. Act.) + .138(St. Act.) 

The F-ratio value for the analysis (F 4 ,io = 26. 1 5, p < .00001). The multiple R = .95538. 

For this set of data, the is .91 and the adjusted R! is .88, so the regression equation is 
able to account for approximately 88% of the variance in the expert ratings. 

Table 4 reports also that the student activities subscale was the only variable which 
could singularly account for the variability in the data (P = .815, t-value = 3.868, t-value 
significant at p = .003 1). This does not necessarily imply that the other variables are not 
contributing factors, because each of the independent variables has a high degree of 
mtercorrelation. 

Limitations 

There are several limitations in this study. Many of these limitations can be accurately 
addressed by further testing and analysis of the Teacher Beliefs Questionnaire and 
Secondary Mathematics Classroom Reform Inventory. 
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Table 3 



Tntercorrelations Between Subscales for Secondary Mathematics Classroom Reform 
Inventory. Teacher Questionnaire, and Expert Ratings 





Subscale 


1 2 


3 


4 


5 


6 


7 


1. 


Expert Ratings 


- .855 

(p<.0001) 


.921 

(.0001) 


.792 

(.0001) 


.757 

(.001) 


.895 

(.0001) 


.907 

(.0001) 


2. 


Curricular Activities 
SMCRI 


- 


.825 

(.0001) 


.841 

(.0001) 


.751 

(.001) 


.947 

(.0001) 


.940 

(.0001) 


3. 


Student Activities 
SMCRI 




- 


.882 

(.0001) 


.635 

(Oil) 


.938 

(.0001) 


.887 

(.0001) 


4. 


Teacher Activities 
SMCRI 






- 


.612 

(.015) 


.958 

(.0001) 


.891 

(.0001) 


5. 


Questionnaire 








- 


.708 

(.003) 


.881 

(.0001) 


6. 


SMCRI Summary 










- 


.958 

(.0001) 


7. 


TQ + SMCRI Sum. 
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Table 4 

Summary of Simultaneous Regression Analysis for Variables Predicting the Expert Rating 
of the Degree of Reform in the Classroom (n = IS) 



Variable 


B 


P 


t 


SiR. t 


Teacher Beliefs Questionnaire 


0.017 


.222 


1.557 


.151 


Teacher Activities 


-0.031 


-.259 


-1.177 


.266 


Curricular Activities 


0.026 


.234 


1.099 


.298 


Student Activities 


0.138 


.815 


3.868 


.003 




Note . Multiple R = .95538; R’ = .912275; Adjusted R' = .87786 
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No attempt was made to select a representative sample of teachers to participate in the 
study. The objective was to purposefully sample in order to maximize variability, so a 
range of classroom environments along the traditional/reformed continuum was necessary. 
Consequently, it is unclear whether the instrument would accurately discriminate among 
classrooms which are narrowly distributed along the continuum. 

Because the instrument was not tested on a sample representative of the population of 
secondary California mathematics classrooms, no statements can be made about the 
general degree of reform evident in the state or the nation today. Many of the questions 
of the Task Force concerning the degree to which the California Mathematics 
Framework's (California Department of Education, 1992) policies have been implemented 
would need to be answered by applying the instrument to a representative sample of 
classrooms. 

Another limitation of the study involved the testing of only 1 5 mathematics teachers 
and classrooms. Statistics performed on this sample may not yield the same conclusions 
when a larger sample is included in the study. A larger sample size of 30+ would offer a 
better perspective from which to study these results. 

The limited number of classroom participants also make it difficult to interpret the 
standardized Beta weights obtained in the statistical analysis. However, standardized Beta 
weights are inherently unstable, and increasing the number of participants is not likely to 
allow for their confident interpretation. 

There is a high degree of correlation between each pair of variables compared in Table 
3. This suggests that multicollinearity between all of the independent variables exists that 
would impair the ability of any one variable to add unique variance to the obtained in 
the regression analysis. While the Student Activities subscale displayed a significant 
weight in the regression equation, it is likely that any combination of the other subscales 
could have predicted the expert rating nearly as well, had Student Activities been omitted 
from the analysis. 

Apart from the means and sample standard deviations found for each of the items, no 
attempt was made to analyze the usefulness of each of the individual items in the Teacher 
Beliefs Questionnaire or Secondary Mathematics Classroom Reform Inventory form, or to 
determine whether they would load along one of the subscales of the video analysis to 
form a validated construct. Performing a factor analysis was not possible because of the 
limited number of participating classrooms. Such a factor analysis could focus the 
instrument on the specific constructs which can discriminate the degree of reform in the 
classroom. 

Some of the experts felt that making a decision about the degree of reform evident 
based on one videotape of one class period was less than desirable. Many of the behaviors 
which might typically be evident in a particular classroom over the course of several 
lessons may have been omitted on the day that the videotaping was done. In addition, one 
of the characteristics of the reformed classroom is multi-day activities. Several of the 
classrooms observed appeared to be in the middle of solving a complex, multi-day 
problem. During such a sequence of class periods, observing only one of them is 
necessarily limiting the nature of the activities exhibited in the sequence. 
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The question of whether a representative sample of a particular classroom was 
obtained by just one “snapshot” is also affected by the presence of the video camera. It 
can be surmised that the presence of the camera affected the classroom environment to 
some degree in most cases. Many of the videos contained footage in which students were 
initially disconcerted by the camera, or asked the instructor questions about the 
videotaping that was being done. 

Several video items were scored as “1” meaning, “not evident at all,” because the 
activity in question was not observed. However, the activity in question could have been 
inferred as normally evident from the general demeanor of the classroom. These item 
scores may have taken away from the summary score for that subscale simply because 
they are items not typically present each day. For example, item #14 on the Teacher 
Activities Subscale judges “How fully does the classroom instructor conduct some form of 
assessment during a learning activity.” While the teacher may typically conduct 
assessment in the context of learning, this may simply not have been the appropriate 
portion of the activity to conduct an assessment. 

Finally, only the researcher used the Secondary Mathematics Classroom Reform 
Inventory form. Therefore, reliability among users of the inventory was never addressed. 
Complications concerning reliability will arise if and when other educators or non-experts 
use the instrument. The instrument’s reliability when used by others will need to be 
investigated to further analyze the instrument’s effectiveness. 

Discussion 

The combination of the Teacher Beliefs Questionnaire score with the three subscales 
of the video analysis yielded a highly predictive regression equation. This analysis 
supports the notion that measuring the degree of reform evident in a secondary 
mathematics classroom is possible. 

Because the scales used in the Teacher Beliefs Questionnaire, the SMCRI and the 
expert ratings are all Likert-type 5 point, with a “1” generally representing “little or no 
evidence” and a “5” representing “fully evident,” comparisons can be made between 
teacher self-reports and independent ratings. The mean of all teachers on the Teacher 
Beliefs Questionnaire was higher than the mean for any of the other variables, including 
the expert ratings, the possibly indicating that teachers are over-reporting where they fall 
into the traditional/reformed continuum. If this were the case, there would be a 
discrepancy between teacher beliefs and behavior in the classroom which would merit 
further investigation. However, it is equally likely statistically that the experts and the 
SMCRI are underreporting the degree of reform in the classroom. While teachers may 
have over-reported their degree of reform in the questionnaire, the high correlation 
between the questiotmaire summary score and the expert rating (r = .757, p <= .001) 
suggests that they accurately r ank themselves relative to their colleagues. The summary 
score for the Teacher Beliefs Questionnaire could stand alone as a valid independent 
variable in predicting the degree of reform in the classroom. This high correlation 
between the questionnaire score and the expert rating suggests that a much less cost 
intensive approach than a video analysis may be effective in determining placement on the 
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traditional/reformed continuum for a classroom environment. This area of investigation 
could yield a cost-effective approach to examining reform. 

Each of the other three independent variables from the SMCRI also had a statistically 
significant degree of correlation with the expert rating. Each of them therefore, could 
stand alone as a valid predictor of the expert rating. While none of them could account 
for the amount of variability that the combination of questionnaire and video subscales 
offered, they are each indicators of the degree of reform evident in the classroom. In fact, 
each variable alone would have accounted for a large portion of the variability in the 
expert ratings: (a) Teacher Beliefs Questionnaire, R! = .5723, (b) SMCRI-Teacher 
Activity subscale, Ri = .6274, (c) SMCRI-Curricular Activity subscale, R! = .7312, and (d) 
SMCRI-Student Activity subscale, R! = .8473. 

The Student Activities Subscale of the SMCRI offered the highest individual 
correlation with the expert rating, and appeared in the regression analysis with statistically 
significant standardized Beta weight. This might suggest that while other activities and 
beliefs may be important in determining reform, ultimately it is what the students are doing 
that offers the best indicator of the classroom environment. The correlation also suggests 
that a colloquial version of the SMCRI, including only the Student Activities subscale, 
could provide an equally valid analysis of the degree of reform in the classroom. 

Future Considerations 

The California Mathematics Task Force Report (California Department of Education, 
1995) recommended that a research system be established to study the implementation and 
effectiveness of the state’ s Mathematics Framework. Task Force members were 
“astonished ... to discover that the state has no means to determine the effectiveness of 
these state policies. The state has collected data on student achievement, but not on the 
processes of teaching and learning that lead to student achievement” (California 
Department of Education, 1995, p. 14). Currently the debate over whether the reforms of 
the Framework are producing higher achievement is alive and well. Those in favor of a 
traditional approach to mathematics education claim that the reform movement has failed. 
Those in favor of the reforms claim that the reforms of the Framework have yet to be 
implemented in most classrooms, and that low achievement scores are therefore reflective 
of the traditional approach to teaching and learning. Indeed, the TIMSS Report found 
that although 8th grade teachers reported familiarity with the NCTM Standards, their 
classroom practices were in general not those of a reformed classroom (U S. Department 
of Education, 1996). The debate has been driven by various political perspectives, still 
with very little substantiating evidence for either side. Yet evidence of the implementation 
and effectiveness of the reforms of the Framework is what is needed. 

The Teacher Beliefs Questionnmre and Secondary Mathematics Classroom Reform 
Inventory provide the vehicle for determining the degree of reform evident in a classroom. 
The instrument can potentially be paired with student achievement data in order to 
determine if the practices, content, and strategies of the Framework are producing 
classrooms shoA^dng higher student achievement. 
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The Mathematics Framework for California Public Schools is currently undergoing 
some revision. One of the charges of the Task Force to the Superintendent of Public 
Schools was to establish rigorous content standards for K-12 mathematics. If significant 
modifications in the Framework are made as a result of this review process, then some 
changes in the measurement instrument may need to be made as well. “Reform” is always 
being redefined, as its definition is contingent on what is considered to be “accepted 
practice.” 

As stated in the literature review, the methods by which student achievement is 
measured is a debated issue. Many questions arise concerning how student achievement 
should be measured. Some educators believe that long-time standard indicators of 
achievement should be used, such as standardized tests and college entrance exams. The 
recent push to include authentic assessment tasks such as portfolio assessment, open- 
ended questions, performance-based assessment, and CLAS-type assessments represents 
another approach to assessment. The type of assessments used will likely be based in part 
upon the content standards developed by the State Superintendent of Public Schools. 

Finally, if a widespread analysis of the number of classrooms exhibiting largely 
reformed environments were performed and few were found, what are the implications for 
the state? Questions should be asked about the readiness of mathematics instructors to 
teach in a reformed manner, and the role and influence of the reform delivery systems. If 
teachers are over-reporting their degree of reform, then that is an indicator that either they 
are in need of more information about what are the specific characteristics of a reformed 
classroom, or they need more practice in self-assessment. Perhaps more attention needs to 
be paid to the type of ongoing inservice training that California teachers are asked to do. 

In conclusion, the pieces are in place for an authentic, research-based look at reform 
and secondaiy mathematics student achievement in California. Rhetoric about whether 
traditional or reformed approaches to teaching are effective can now be replaced by a 
meaningful research-based look at which classrooms are traditional, and which are 
reformed. Then studies comparing teaching methods and student achievement can be 
undertaken to provide an accurate picture of the condition of mathematics education in 
California. Perhaps then we can each cast aside our naive assumptions and construct a 
model of the learning of mathematics that can be useful for every student in California’s 
public schools. 
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